Artificial neural networks have utility in a wide variety of computing environments, such as speech recognition, process control, optical character recognition, handwriting recognition, continuous logic or fuzzy logic, engineering and scientific computations, signal processing, and image processing. Processing engines for many of the foregoing computing environments may be implemented through neural networks comprising a plurality of elemental logic elements called neuron circuits.
A neuron circuit (or processing element) is the fundamental building block of a neural network. A neuron circuit has multiple inputs and one output. The above-identified Related Invention No. 1 discloses, in one embodiment, a neuron circuit which comprises only an adder as its main processing element.
As discussed in Related Invention No. 2 identified above, there are more than twenty known types of neural network architectures, of which the "back-propagation", "perceptron", and "Hopfield network" are the best known. Related Invention No. 2 discloses, in one embodiment, a neural network which does not require repetitive training, which yields a global minimum to each given set of input vectors, and which has an architecture that comprises three layers.
The three layers of Related Invention No. 2 identified above are known as the "input layer", the "hidden layer", and the "output layer". The function of the input layer is to receive a plurality of neural network inputs and then distribute them to the inputs of a plurality of neuron circuits contained in the hidden layer. The hidden layer comprises a plurality of neuron circuits. Each neuron circuit in the hidden layer may perform arithmetic operations on its inputs to produce a corresponding neuron circuit output. The output layer receives as inputs the outputs of the plurality of neuron circuits and sums them to produce neural network outputs.
It is thus a primary function of the output layer to sum the outputs of the neuron circuits contained in the hidden layer. Various output-processing circuits for neural networks are known in the art, including some which utilize output layers to sum the outputs of the hidden layer.
FIG. 1, for example, is a block diagram of a prior art digital adder chain for producing a neural network output. The outputs of neuron circuits 1 and 3 are coupled to adder circuit 5, and the outputs of neuron circuits 7 and 9 are coupled to adder circuit 11. The outputs of adder circuits 5 and 11 are in turn coupled to adder circuit 13, which produces the neural network output y.
The adder chain shown in FIG. 1 suffers a serious drawback, in that it consumes a large amount of space when implemented on an integrated circuit. The reason for this is that digital adder circuits are limited to receiving only two operands at a time. Consequently, the number of adder circuits required in an adder chain is N-1, where N is the number of neuron circuits that provide inputs to the adder chain. Furthermore, a neural network output-processing circuit producing a plurality of outputs would require one adder chain per output. In other words, a typical neural network output-processing circuit using an adder chain requires a large number of adder circuits and thus consumes a vast amount of silicon space when implemented on an integrated circuit.
FIG. 2 is a block diagram of a prior art multiple-RISC (Reduced Instruction Set Computer) circuit for producing a neural network output. In the circuit shown in FIG. 2, the outputs of neuron circuits 20, 22, 24, and 26 are sequentially broadcast across bus 27 to RISC 32 and RISC 34. Each RISC is responsive to a predetermined number of the neuron circuit outputs and sums these outputs to produce a corresponding neural network output. The output 36 of RISC 32 is neural network output y.sub.1, and the output 38 of RISC 34 is neural network output y.sub.2.
Unfortunately, the approach shown in FIG. 2 requires the use of at least one RISC. RISC's consume large integrated circuit areas and are expensive. Furthermore, RISC's used in a neural network output layer require programming which is difficult and time-consuming.
FIG. 3 is a block diagram of a prior art circuit, using a single RISC circuit and a plurality of adder circuits, for producing neural network outputs. In the circuit shown in FIG. 3, the outputs of neuron circuits 50, 52, 54, and 56 are sequentially broadcast across bus 57 to RISC 60. RISC 60 is connected to addressable latches 68 and 70 via bus 63. Upon receiving a neuron circuit output on bus 57, RISC 60 writes the neuron circuit output to one or both of the addressable latches. RISC 60 can write to only one of the latches at a time across bus 63. RISC 60 is programmed to determine which latch receives which neuron circuit output and in what order the latches receive the neuron circuit outputs. The output of latch 68 is connected to a first input of adder circuit 76 via line 72. Adder circuit 76 sums neuron circuit outputs received by latch 68. The output of adder 76 is connected via line 78 to a second input of adder 76. Adder 76 updates its output 84 when RISC 60 writes to latch 68. The output 84 of adder circuit 76 represents neural network output y.sub.1.
Likewise, the output of latch 70 is connected to a first input of adder circuit 80 via line 74. Adder circuit 80 sums neuron circuit outputs received by latch 70. The output of adder 80 is connected via line 82 to a second input of adder 80. Adder 80 updates its output 86 when RISC 60 writes to latch 70. The output 86 of adder circuit 86 represents neural network output y.sub.2.
The approach shown in FIG. 3 suffers the same drawbacks as the prior art circuit shown in FIG. 2. The RISC consumes a large integrated circuit area and is expensive. Furthermore, the RISC requires programming which is difficult, time-consuming, and expensive.
Therefore, there is a significant need for a neural network output-processing circuit which has a straight-forward architecture and which is easy and inexpensive to implement.